It’s late night Sunday and rather than bringing you exciting news from the world of search and SEO, I’ve got some explaining to do. For those who hadn’t noticed, SEOmoz has some serious downtime and errors this weekend. Starting early morning Saturday and running through to Sunday, many parts of the site were inaccessible due to either A) an exceptionally unlucky set of simultaneous hardware/software failures on our host and backup servers, or B) the act of a vengeful Norse god (Odin, we’re looking in your direction).
In any case, this catastrophe was exacerbated due to our recent hosting move – whenever you’re shifting host locations, there’s a certain amount of finger crossing to be done, particularly with relation to data backups. Jeff & Mel did a great job here, but this unlucky strike had a few casualties which couldn’t be recovered.
- Most significant was the loss of the last two weeks of stored data from the Rank Tracker tool. We will again migrate all rankings saved before the Rank Tracker launch, but rankings and settings stored after the launch are not retrievable. We cannot apologize enough for this loss and have taken steps to ensure that this will not happen again.
- The Rank Tracker tool itself is still down while the migration completes, but should be back up in the next 24 hours. When the service returns, you will find significantly increased limits for manual rankings run per day and for automated rankings as well.
- A smaller data loss may be noted for saved Linkscape advanced reports. Some reports run in the last day or so may not have been saved. We have added 5 Linkscape credits to all accounts to accommodate this.
- Blog posts, user profiles, Q+A questions, marketplace profiles and any other database-reliant content created between Saturday afternoon (when the site briefly recovered) and Sunday midday (when our last round of failures/attacks occurred) is irreparably lost. Again, we’re taking steps to ensure that backups will be secure and solid for the future, hopefully making this a one-time only event.
- Other fragments of data, including some Q+A questions, blog post & YOUmoz images, profile pictures and other database elements from the past 2 weeks may also have suffered. We’ve recovered nearly everything (and possibly got all of them), but if you notice some oddities, this is most likely why.
For PRO members, we’re doing our best to make up this weekend’s events to you with greater access to those tools hit hardest. We’ll also work tirelessly this week (despite the SMX Advanced conference in Seattle) both to prevent future mishaps like this and to provide a high level of support for anything you need.
To our readers, visitors and regulars – I offer my personal apology. I know that you like to read and use SEOmoz on the weekends, and I’m sure this interfered with your regular course of business. We’ve got a great team of developers here at SEOmoz, and I’m incredibly proud of their performance under fire over the last 48 hours. Rest assured that all of us will put our shoulders to the wheel to make data integrity and uptime priorities over the weeks and months to come.
Thanks for your patience and understanding,
Rand Fishkin, CEO
p.s. Our thanks also to ex-mozzer Jane Copland, whose quick eye and catlike reflexes over IM brought the site issues to the attention of our dev team very early on and probably saved us additional heartache.
p.p.s. A few other areas are affected – Labs, Q+A Search & our RSS feed. We’re working to get these back online today as well. If you find anything else, please post in the comments; we appreciate all the help!
UPDATE 2: We’re going to go down for a couple hours around 1pm Pacific, 4pm Eastern today in order to secure our data in additional locations. We hope to be back up by 3-4pm Pacific this afternoon. You can follow the SEOmoz Twitter account for the latest on this front.
